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As  you  may  know,  to  help  BSO  do  a  better  job  of  meeting  agencies'  data  pro- 
cessing needs,  I  am  conducting  regular,  periodic  interviews  with  BSO  users.   A 
report  on  the  first  set  of  interviews,  which  took  place  during  the  first  quarter 
of  FY»84,  is  enclosed. 

This  spring,  in  addition  to  conducting  further  interviews,  I  will  be  work- 
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I  hope  that  you  find  this  report  of  interest  and  of  use.  Do  let  me  know 
your  reactions. 
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INTRODUCTION 

What  follows  is  an  interim  report  of  the  User  Satisfaction  Project,  an  ongoing 
initiative  of  the  Office  of  Management  Information  Systems  (OMIS).  This  report 
summarizes  project  activities  to  date,  and  presents  findings  on  the  views  and 
experiences  of  individuals  at  twelve  state  agencies  that  use  information  pro- 
cessing services  provided  by  the  Bureau  of  Systems  Operations  (BSO). 

Organization  Overview 

The  Office  of  Management  Information  Systems  (OMIS),  part  of  the  Commonwealth  of 
Massachusetts'  Executive  Office  of  Administration  and  Finance,  was  established 
in  1979  by  an  Executive  Order.  OMIS  is  responsible  for  ensuring  optimum  utili- 
zation, maintenance  and  control  of  information  processing  systems  established 
and  owned  by  the  Commonwealth.  Information  processing  is  defined  to  include 
both  data  processing  and  office  automation  equipment  such  as  word  processing, 
micro-computers,  and  electronic  mail. 

OMIS  is  organized  into  two  major  divisions,  the  Bureau  of  Systems  Operations 
(BSO)  and  the  Bureau  of  Systems  Policy  and  Planning  (BSPP). 

BSO  operates  a  major  data  center  that  serves  approximately  60  state  government 
organizations,  including  Administration  and  Finance  and  Human  Services  agencies, 
and  the  Legislature.   The  Bureau's  mandate  is  to  deliver  timely,  efficient  and 
economically  pragmatic  solutions  to  meet  the  Commonwealth's  information  systems 
needs.  BSO  has  a  family  of  IBM  and  IBM-compatible  mainframes  that  support  over 
1,500  on-line  devices.   BSO  also  performs  software  maintenance  and  system  deve- 
lopment activities  for  many  of  its  users. 

BSPP  is  charged  by  Executive  Order  and  statute  with  oversight  of  all  infor- 
mation processing  procurements  for  the  Commonwealth.   The  Bureau's  primary 
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Qrganization  Overview  (continued) 

objective  is  to  assist  agencies  in  the  implementation  of  automated  information 
systems  that  will  enable  them  to  perform  more  efficiently  and  effectively.  Its 
responsibilities  include  helping  state  agencies  and  state  data  centers  (of  which 
BSO  is  one)  to  develop  data  processing  plans,  and  preparing  a  statewide  data 
processing  plan.  BSPP  provides  assistance  to  agencies  in  the  procurement  of 
data  processing  equipment,  systems,  and  services,  and  in  defining  data  pro- 
cessing needs. 

BSO  and  BSPP  are  supported  by  two  other  units,  the  Office  of  Administration  and 
the  Office  of  General  Counsel. 

Project  Overview 

The  BSO  User  Satisfaction  Project  began  late  in  Fiscal  Year  1983  as  an  ini- 
tiative of  the  Director  of  OMIS.  The  Project  entails  determining  users'  percep- 
tions of  the  quality  of  BSO  services,  identifying  common  problem  areas,  and 
developing  responsive  strategies  to  resolve  these  problems.  The  Project's 
overriding  purpose  is  to  support  OMIS  in  carrying  out  its  service  mission. 

The  Project,  which  is  intended  to  promote  organizational  development,  was 
designed  as  an  applied  research  effort,  in  which  information  on  users'  satisfac- 
tion with  BSO  would  be  obtained  both  through  interviewing  and  through  admi- 
nistering a  questionnaire  to  appropriate  personnel  at  agencies  that  receive  BSO 
services.  The  questionnaire  was  intended  to  provide  quantitative  information 
about  individuals'  perceptions  of  BSO  services  in  sufficient  detail  to  isolate 
problem  areas.  The  interview  was  intended  to  promote  dialogue  between  a  repre- 
sentative of  OMIS  and  representatives  of  agencies  using  BSO  services,  and  to 
obtain  qualitative  data  on  these  individuals'  experiences  that  would  be  of  use 
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Project  Overview  (continued) 

in  interpreting  and  responding  to  the  questionnaire  results.  Research  methods 
were  employed  so  that  any  changes  in  BSO  users'  satisfaction  over  time  could  be 
reliably  determined. 

Data  Collection:  First  Quarter  FY' 84 

During  the  first  quarter  of  FY1 84,  group  interviews  were  conducted  with  a  total 
of  forty  nine  participants  from  twelve  agencies.   (A  listing  of  these  agencies 
appears  in  Table  3  of  Appendix  C.)  Group  size,  four  on  average,  ranged  from  two 
to  six  participants.   The  interviews  took  about  two  hours  and  included  time 
during  which  the  questionnaire  on  users'  perceptions  of  BSO  services  was 
administered;  questionnaires  were  completed  by  forty  five  of  the  participants 
from  eleven  of  the  agencies. 

All  interviews  were  conducted  by  the  Special  Assistant  to  the  Director  of  OMIS. 
In  one  case,  at  the  suggestion  of  the  agency,  an  analyst  from  the  Bureau  of 
Systems  Policy  and  Planning  (BSPP),  who  recently  had  been  assigned  to  work  with 
that  agency,  sat  in  on  the  interview. 

Methodology 

Inevitably,  a  number  of  methodological  issues  have  emerged  during  the  course  of 
designing  and  implementing  the  project.  Because  this  interim  report  is  intended 
as  a  management  document,  only  the  most  directly  relevant  methodological  con- 
cerns are  addressed,  here.   These  include  questionnaire  and  interview  instrument 
development  and  data  analysis  procedures,  which  are  discussed  in  Appendix  A. 
Overall  project  methodology  will  be  reviewed  in  greater  detail  in  a  final  report 
on  the  Project. 


FINDINGS 

In  reviewing  the  results  reported  below,  it  is  important  to  remember  that  they 
were  derived  from  data  gathered  during  the  first  quarter  of  Fiscal  Year  1984. 
(Data  collection  and  analysis  procedures  are  summarized  in  Appendix  A.)  These 
results  cannot  be  assumed  to  reflect  the  current  level  of  satisfaction  with  BSO, 
even  among  those  who  participated  in  the  interviews  conducted  last  July,  August 
and  September.  Any  number  of  changes  within  the  participating  user  agencies  and 
within  BSO  may  have  intervened  to  alter  these  users'  perceptions  of  BSO  ser- 
vices. 

In  fact,  the  User  Satisfaction  Project  was  launched  as  one  of  several  efforts  by 
the  Director  of  OMIS  to  increase  BSO's  responsiveness  to  its  user  community,  and 
some  of  the  interview  comments  suggested  that  those  efforts  were  being  felt  even 
as  these  interviews  were  being  conducted.  Furthermore,  the  project  is  intended 
to  promote  organizational  development;  therefore,  when  feasible,  problems  that 
were  identified  during  an  interview  were  brought  at  once  to  the  attention  of 
appropriate  OMIS  personnel. 

To  determine  the  extent  to  which  BSO  has  been  successful  in  responding  to  these 
problems  and,  more  generally,  the  extent  and  direction  of  any  change  in  users' 
satisfaction  with  BSO  services  will  require  a  follow-up  inquiry. 

Agency  and  Inter-Agency  Satisfaction 
Questionnaire  Results  by  BSO  Service  Category: 

The  User  Satisfaction  Project  questionnaire  measures  satisfaction  in  six  BSO 
service  categories.  These  categories  are  1)  technical  assistance  in  problem 
determination  and  response,  2)  technical  assistance  in  teleprocessing  applica- 
tions, 3)  technical  assistance  in  data  base/file  management,  M)  batch  pro- 
cessing, 5)  on-line  processing,  and  6)  maintenance/enhancement  of  applications. 


-5- 

Agency  and  Inter-Agency  Satisfaction  (continued) 

(Definitions  of  these  categories  appear  in  Table  1  of  Appendix  B.)  Tables  4  and 
5  in  Appendix  C  present  each  agency's  mean  satisfaction  rating  for  each  category 
as  well  as  the  grand  mean  satisfaction  score  for  each  category.  As  these  tables 
show,  questionnaire  respondents  tended  to  be  more  satisfied  than  not  with  the 
services  they  received  from  BSO.  Grand  mean  satisfaction  scores  for  all  six 
categories  are  to  the  left,  or  satisfied  side  of  four,  the  neutral  position  on 
the  seven  point  rating  scale.  Questionnaire  responses  indicate  participants' 
slight  satisfaction,  overall,  with  the  six  BSO  services  surveyed. 

Interview  Findings: 

On  average,  the  grand  mean  satisfaction  scores  discussed  above  are  marginally 
better  than  the  more  general  measure  of  overall  satisfaction  with  BSO  obtained 
during  each  group  interview.  By  contrast  with  questionnaire  results,  three  of 
the  groups  interviewed  gave  ratings  to  the  right,  or  dissatisfied  side  of  the 
neutral  position  on  the  rating  scale.  Even  so,  the  simple  average  of  the 
ratings  obtained  in  these  interviews  is  3»7,  just  to  the  left,  or  satisfied  side 
of  neutral. 

A  number  of  concerns  were  recurrently  voiced,  and  clearly  caused  dissatisfaction 
(and  frustration)  among  user  participants.  Those  most  frequently  mentioned  are 
summarized  below.  The  order  in  which  they  are  presented  is  not  intended  to 
indicate  their  relative  importance  to  users. 

a.  Inadequate  capacity  planning.  Although  a  number  of  participants  cited 
BSO's  acquisition  of  the  Amdahl  as  an  important,  positive  event,  they 
remained  fearful  that  the  pre-Amdahl  situation  would  recur.  They  were 
aware  that  "capacity  planning"  was  now  part  of  BSO's  vocabulary,  but 
many  expressed  skepticism  about  the  effectiveness  of  the  BSO  capacity 
planning  function. 
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Agency  and  Inter-Agency  Satisfaction  (continued) 

b.  Paucity  of  qualified  staff.  Although  repeated  references  were  made  to 
BSO's  able  employees,  many  participants  expressed  frustration  with  BSO's 
lack  of  depth  in  staff  qualified  to  meet  user  agencies'  technical 
assistance  needs.  "We  need  a  service  bureau  all  day  long,  not  just 
until  one  technical  expert  goes  home  at  3  p.m.,"  was  a  typical  comment. 
Many  participants  expressed  concern  about  the  extent  to  which  the 
Commonwealth's  salary  structure  and  civil  service  requirements 
constrain  efforts  to  recruit  and  retain  qualified  data  processing 
professionals.  Some  believed  that  passage  of  the  Technical  Pay  Bill 
would  address  this  concern,  but  others  were  skeptical  about  the  bill's 
potential  for  longterm  effectiveness. 

c.  Poor — or  poorly  communicated — organizational  structure.  Individuals 
frequently  associated  this  concern  with  their  views  of  BSO  staff 
(described  in  b.  above).  Some  thought  that  competent  BSO  staff  seemed 
handicapped  by  organizational  constraints;  some  expressed  frustration 
with  the  difficulty  they  experienced  when  attempting  to  contact  the 
correct  BSO  employee,  or  even  to  identify  that  employee,  in  order  to 
obtain  technical  assistance  or  same  other  service. 

d.  Insufficient,  ineffective  procedural/paperwork  requirements.  Par- 
ticularly in  light  of  BSO's  growth,  no  one  took  issue  with  the  need  for 
procedures,  and  some  even  applauded  the  Bureau's  efforts  to  establish 
management  (as  distinguished  from  information)  systems.  However,  a 
number  of  participants  perceived  existing  procedures,  like  BSO's  organi- 
zational structure,  to  be  poorly  conceived,  poorly  communicated,  or 
both.   Some  participants  expressed  the  view  that  paperwork  requirements 
were  operator-  rather  than  user-oriented.  A  sample  comment  about  BSO, 
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"They  make  us  complete  a  morass  of  paperwork;  we  never  get  anything  in 
writing  from  them,"  conveys  the  frustration  that  some  user  participants 
felt  about  this  issue. 

e.  Poor  internal  communication.   Participants  observed  poor  communication 
between  BSO  staff  on  the  8th  and  the  16th  floors,  between  the  day  and 
night  shifts,  and  between  state  personnel  and  "03"  consultants,  and 
felt  that  poor  communication  in  any  of  these  three  cases  jeopardized 
delivery  of  effective  service. 

f.  Insufficient  collaboration  with  users.  In  addition  to  communication 
problems  associated  with  organizational  structure  and  procedures  (c. 
and  d.  above),  many  participants  recalled  situations  in  which  they  had 
not  been  consulted  about  or  even  notified  of  an  action  that  seriously 
affected  them  until  after  BSO  had  taken  that  action.  Some  of  those 
interviewed  were  concerned  that  BSO  staff  tended  to  be  defensive  and 
seemed  unwilling  to  recognize  or  draw  upon  the  competence  of  user 
agencies'  DP  staff.  Some  users  expressed  regret  that  BSO  staff  were 
not  more  familiar  with  the  workings  of  user  agencies. 

In  addition,  participants  were  asked  specifically  about  the  BSO  User 

Group  meetings.   Their  responses  to  this  inquiry  are  presented  below. 

1 
BSO  User  Group  Meetings 

Interview  Findings: 

Group  ratings  on  satisfaction  with  BSO  User  Group  Meetings  were  solicited  during 

the  interviews.  By  contrast  with  both  questionnaire  results  and  group  interview 

ratings  of  overall  satisfaction  with  BSO,  these  interview  ratings  indicate  that 


1 
These  are  quarterly  meetings  conducted  by  BSO  and  should  not  be  confused  with 
either  the  EOHS  Data  Policy  Group  meetings  or  the  PMIS  User  Group  meetings. 
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user  participants  tended  to  be  more  dissatisfied  than  not  with  BSO  User  Group 
meetings.  Only  one  agency  gave  a  rating  to  the  left,  or  satisfied  side  of  four, 
the  neutral  position  on  the  rating  scale.  The  simple  average  of  the  ratings 
given  is  5.0,  one  position  to  the  right,  or  dissatisfied  side  of  neutral.   The 
most  frequently  mentioned  problems  that  participants  had  with  the  BSO  User  Group 
meetings  are  summarized  below.  The  order  in  which  they  are  presented  is  not 
intended  to  indicate  their  relative  importance  to  users. 

a.  Lack  of  Interest  in  Users'  Problems.  Although  many  of  those  inter- 
viewed acknowledged  that  users  are  regularly  invited  to  air  their  con- 
cerns at  these  meetings,  a  number  of  participants  questioned  BSO 
representatives'  real  interest  in  users'  problems.  Responses  were 
described  by  some  as  glib  or  defensive;   "BSO  just  gives  lip 
service.  Our  problems  get  explained  away."  Both  the  infrequency 

of  the  meetings  (quarterly)  and  the  erratic  attendance  of  BSO  managers 
and  technical  staff  were  believed  by  some  to  reflect  BSO's  essential 
lack  of  interest  in  their  users. 

b.  Wrong  Forum  for  Problem  Solving.  A  number  of  participants  felt  that 
the  meetings  were  a  good  source  of  general  information  about  BSO,  but 
that  they  were  too  large  and  too  public  to  be  an  effective  forum  for 
problem-solving.  Some  felt  that  to  discuss  one  agency's  problem  fully 
would  be  an  imposition  on  users  from  other  agencies,  or  that  it  would 
be  "airing  dirty  linen."  Several  individuals  reported  feeling  intimi- 
dated; they  were  hesitant  to  say,  in  this  forum,  that  they  did  not 
understand  the  technical  jargon  used  by  BSO  staff.  (Some  questioned 
whether  overly  technical  responses  weren't  a  deliberate  strategy  by  BSO 
staff  to  avoid  real  discussion  of  users'  problems.  See  a.  above.) 
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BSO  User  Group  Meetings  (continued) 

c.  Lack  of  Planning.  Some  participants  felt  that  an  inadequate  effort  was 
made  to  plan  the  meetings.  A  number  of  individuals  said  that  they  were 
only  notified  about  upcoming  meetings  at  the  last  minute,  and  sometimes 
only  by  word  of  mouth.  Meeting  agendas  were  not  always  disseminated 

or  apparently  even  set  in  advance;  none  of  the  participants  reported 
ever  having  been  involved  in  planning  an  agenda. 

d.  Lack  of  Follow-Through.   In  addition  to  BSO's  apparent  lack  of  interest 
in  users'  problems  during  the  meetings  (see  a.  above)  a  number  of  parti- 
cipants commented  on  BSO's  lack  of  subsequent  responsive  action.  Users 
reported  receiving  no  minutes  of  the  User  Group  meetings.  Most  par- 
ticipants saw  little  effort  on  the  part  of  BSO,  as  a  result  of  these 

•  meetings,  to  develop  responsive  policies. 

Relationship  between  Overall  Usage  and  Satisfaction 
Questionnaire  Results  by  BSO  Service  Category: 

On  the  theory  that  a  service  bureau  would  devote  its  attention  primarily  to 
meeting  the  needs  of  its  major  customers,  questionnaire  data  were  used  to  deter- 
mine whether  or  not  greater  consumers  of  BSO  services  would  be  more  satisfied 

than  would  lesser  consumers.  The  cost  of  BSO  services  allocated  to  each  agency 

1 
for  Fiscal  Year  1983  was  used  as  a  measure  of  that  agency's  usage. 

A  set  of  analyses  performed  to  establish  whether  the  amount  of  an  agency's  BSO 
usage  was  associated  with  that  agency's  satisfaction  with  BSO  services  produced 
two  statistically  significant  results.  First,  satisfaction  with  technical 
assistance  in  teleprocessing  applications  is  negatively  correlated  with  overall 
usage.  Second,  satisfaction  with  BSO's  batch  processing  services  is  positively 
correlated  with  overall  usage.  These  results  appear  in  Table  6  of  Appendix  C. 

1 

These  figures  were  obtained  from  "OMIS  Chargeback  System,"  a  report  required  by 
the  Commonwealth  of  Massachusetts,  1983  Acts  and  Resolves,  Chapter  289,  line 
item  1101-2300. 
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At  the  end  of  each  group  interview,  participants'  suggestions  and  requests  were 

solicited.  Responses  have  been  topically  ordered,  and  are  presented  below  as 

they  were  recorded. 

Collaboration: 

"Recognize  users'  experience  and  expertise.  Let  us  feel  that  if  we  make 

constructive  suggestions,  you'll  listen." 

Interagency  Agreement: 

"Executive  agreements  that  have  substance,  are  realistic,  and  have  accoun- 
tability both  ways." 

Newsletter: 

"We'd  really  like  to  see  a  BSO  Users'  newsletter." 

"What  about  a  BSO  newsletter?  Through  electronic  mail?" 

Notification  of  Changes: 

"Changes  should  be  communicated  by  memo  and  over  TSO." 

"Please  notify  us  in  advance  of  any  changes." 

"If  you're  considering  a  change  that's  going  to  affect  me,  can  we  talk  about  it 

before  you  implement  it?" 

Technical  Assistance: 

"It  would  be  very  helpful  if  BSO  had  a  set  of  specialists  to  whom  agency  people 

could  go." 

"A  Help  Desk  would  be  a  revolutionary  improvement." 

Training  and  Education: 

"BSO  is  getting  better  at  providing  training  for  its  users.   That  should  be 

continued ." 
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Agency  Participant's  Suggestions  (continued) 

"I'd  like  to  see  user  agencies'  top  management  receive  education  about  BSO,  so 

that  [user  DP  personnel]  don't  get  caught  in  the  middle." 

"I'd  like  to  see  an  orientation  to  BSO  for  our  clerical  staff — a  tour  and  a 

briefing — a  simple  explanation  of  BSO's  hardware  and  of  how  [the  clerical 

staff's]  work  relates  to  what  BSO  does." 

"Our  programmers,  who  live  at  BSO,  should  be  eligible  for  any  technical  training 

you  provide.  We'd  pay." 

"Train  your  users  to  be  good  users." 

User-Friendly  Procedures: 

"There  needs  to  be  a  way  to  get  disk  storage  without  buying  a  whole  disk  drive 

or  having  to  beg." 

"Is  there  any  way  to  cut  back  on  BSO's  paperwork  requirements?" 

"Set  up  a  counter  where  users  can  walk  in  and  get  their  jobs  stamped  in." 

"If  there's  a  problem  with  a  job,  please  let  me  know  right  away,  and  let  me  know 

what  I  have  to  do." 

"We  should  have  a  formally  designated  contact  person  in  BSO;  we  rely  on  informal 

contacts." 

User  Group  Meetings: 

"Revamp  these  meetings." 

"There  should  be  smaller  groups  meeting  more  frequently." 

"Let  us  know  the  agenda  ahead  of  time." 

"Send  us  minutes  of  the  meetings." 

"Don't  use  technical  jargon." 

"People  should  introduce  themselves;  wear  name  tags." 

"This  should  be  an  opportunity  for  relationship-building." 
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INTERPRETATIONS  AND  SUGGESTIONS 

In  this  section,  some  interpretations  of  findings  about  users'  satisfaction  with 
BSO  services  are  offered;  these  are  followed  by  suggestions  for  possible  responses. 

Agency  and  Inter-Agency  Satisfaction 

Recall  that  slight  differences  emerged  between  agencies'  questionnaire  and  group 
interview  satisfaction  ratings.  While  both  sets  of  ratings  suggest  that  on 
average,  users  who  participated  in  this  project  were  slightly  more  satisfied 
with  BSO  services  than  not,  interview  ratings  tended  to  be  less  favorable  than 
questionnaire  ratings.  On  average,  interview  ratings  fell  very  close  to  the 
neutral  position  on  the  rating  scale,  and  several  agencies'  interview  ratings 
indicated  overall  dissatisfaction  with  BSO. 

Although  there  is  no  explanation  that  indisputably  accounts  for  differences  bet- 
ween questionnaire  and  interview  responses,  two  possible  factors  warrant  mention 
here. 

First,  the  User  Satisfaction  Project  questionnaire  measures  satisfaction  in 
terms  of  six  service  categories  that  OMIS  senior  management  identified  as  impor- 
tant.   It  may  be  that  user  agency  participants,  when  rating  their  satisfaction 
with  BSO  services  during  group  interviews,  took  into  account  other  services  than 
those  specified  in  the  questionnaire.  Interview  ratings  on  satisfaction  with 
the  BSO  User  Group  meetings  support  this  contention. 

Second,  while  statistical  analyses  of  questionnaire  responses  treated  each 
participant's  ratings  equally,  in  the  group  interviews  some  individuals  may  have 
deferred  to  others  during  the  process  of  rating  satisfaction  with  BSO. 
Particularly  in  light  of  time  constraints  for  reaching  consensus  necessitated  by 
the  limits  of  the  interview,  the  considerable  intra-agency  variance  in  question- 
naire responses  supports  this  contention. 


Inter-Agency  Differences 

According  to  both  questionnaire  and  group  interview  responses,  the  Office  of  the 
State  Auditor  was  considerably  more  dissatisfied  with  BSO  services  than  were 
participants  from  other  agencies.   (Tables  H   and  5  in  Appendix  C  show  what  sta- 
tistical analyses  prove,  that  the  questionnaire  responses  of  the  Office  of  the 
State  Auditor  differed  significantly  from  the  responses  of  the  other  agencies.) 

In  reflecting  upon  these  findings,  it  is  worth  noting  that,  among  participating 
agencies,  the  Office  of  the  State  Auditor  is  the  newest  member  of  the  BSO  user 
community.  Furthermore,  unlike  many  other  BSO  users,  the  Auditor's  staff  sought 
services  directly  from  BSO  Operations,  rather  than  first  working  with  BSO 
Systems  &  Programming  staff.  At  the  time  of  this  data  gathering  effort,  BSO 
Operations  had  no  formal  orientation  for  new  users,  and  most  project  par- 
ticipants who  reported  having  a  formal  BSO  contact  person  identified  a  member  of 
the  Systems  &  Programming  staff.  During  their  group  interview,  the  questions 
and  comments  of  participants  from  the  Office  of  the  State  Auditor  demonstrated 
these  users'  need  for  some  form  of  introduction  to  BSO  Operations,  and  for  a 
contact  person  within  BSO  to  whom  they  could  address  their  questions  and  from 
whom  they  could  expect  a  response. 

BSO  User  Group  Meetings 

As  reported  in  the  Findings  section,  interview  participants  tended  to  be  more 
dissatisfied  than  not  with  the  BSO  User  Group  meetings.  These  meetings  received 
less  favorable  ratings  than  did  BSO  overall.  In  reflecting  upon  these  scores, 
it  seems  worthwhile  to  consider  the  assumption,  implicit  in  many  of  the 
participants'  associated  comments,  that  these  meetings  belong  to  and  are  the 
responsibility  of  BSO,  rather  than  being  users'  group  meetings. 
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Relationship  between  Overall  Usage  and  Satisfaction 

Analysis  of  questionnaire  data  revealed  that  agencies'  satisfaction  with  tech- 
nical assistance  in  teleprocessing  applications  is  negatively  correlated  and 
satisfaction  with  batch  processing  is  positively  correlated  with  usage.  In 
interpreting  these  findings,  it  should  be  noted  that  although  statistically 
significant,  the  statistical  relationship  between  variables  (usage  and  satisfac- 
tion category)  is  relatively  weak  in  both  of  these  cases.  Furthermore,  BSO 
usage  was  measured  globally;  within  each  agency's  overall  expenditure  alloca- 
tion, no  attempt  was  made  to  discriminate  among  various  types  of  usage.   Under 
the  circumstances,  it  would  be  premature  to  draw  practical  conclusions  from 
these  results  at  this  time.  However,  it  might  prove  useful  to  conduct  a  more 
detailed  assessment  in  conjunction  with  implementation  of  the  planned  chargeback 
system. 

General  Suggestions 

In  addition  to  the  specific  suggestions  of  project  participants  presented  in  the 

Findings  section  of  this  report,  several  general  suggestions  are  offered  here. 

1.  Review  all  concerns  documented  in  this  report,  that  user  participants 
identified  in  their  interview  comments  and/or  through  satisfaction 
ratings. 

2.  Develop  an  understanding  of  each  concern,  in  sufficient  depth  to  ensure 
the  suitability  of  any  subsequently  planned,  responsive  action. 

3.  In  developing  such  understanding  and  in  developing  responses,  draw  upon 
the  experience  and  expertise  of  members  of  the  user  community. 

4.  Keep  users  informed  of  activities  designed  to  enhance  BSO  service 
delivery;  in  particular,  keep  them  up  to  date  concerning  the  progress  of 
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General  Suggestions 

activities  designed  to  address  their  expressed  concerns.   (For  example, 
users  should  be  informed  regularly  about  the  status  of  the  Technical  Pay 
Bill.  Recent  comments  by  some  who  participated  in  the  first  quarter 
FY '84  interviews  indicate  that  they  have  no  knowledge  of  current  imple- 
mentation planning  activities.) 

5.  Establish  methods  by  which  users'  new  and  recurring  concerns  will  be 
brought  to  the  attention  of  appropriate  OMIS  staff  on  a  regular  basis, 
and  will  be  addressed  as  outlined  above. 

6.  Conduct  periodic  data  collection  to  determine  the  extent  and  direction 
of  any  change  in  users'  satisfaction  with  BSO  services. 
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Appendix  A 


1.  Questionnaire  Development. 

2.  Interview  Instrument  Development 

3.  Questionnaire  Data  Analysis. 

4.  Interview  Data  Analysis. 
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Questionnaire  Development 

A  literature  search  conducted  at  the  end  of  Fiscal  Year  1983  unearthed  com- 
paratively few  articles  that  focus  on  computer  user  satisfaction;  most  published 
research  in  the  area  is  devoted  to  the  technical  side  of  EDP.  The  lack  of  cri- 
tical reviews  of  the  literature  is  indicative  of  the  relative  infancy  of  this 
field.  Only  one  article  that  was  located  dealt  specifically  with  the  concept  of 
user  satisfaction  (others  were  concerned  with  the  relationships  between  user 
behavior,  user  involvement,  user  attitudes  and  user  background);  a  brief 
discussion  of  the  measurement  instrument  developed  by  the  authors  of  that 
article,  and  subsequently  modified  for  use  in  the  User  Satisfaction  Project, 
appears  as  Appendix  A  to  this  report. 

1 
Bailey  and  Pearson  developed  the  most  comprehensive  concept  of  user  satisfac- 
tion, and  also  devoted  more  attention  to  reliability  and  validity  than  was 
reported  in  other  articles.   Realizing  that  there  was  no  comprehensive, 
generally  accepted  definition  of  user  satisfaction,  these  researchers  attempted 
to  identify  every  aspect  of  computer  user  satisfaction.  After  conducting  an 
extensive  literature  search  of  their  own  (which  included  reviews  of  work  by  both 
psychologists  and  EDP  researchers),  and  interviewing  data  processing  pro- 
fessionals and  middle  manager  users,  Bailey  and  Pearson  identified  39  factors 
relating  to  computer  user  satisfaction.  Based  on  a  statistical  analysis,  they 
reported  a  0.99  probability  that  this  was  a  complete  list  of  the  factors  that 
comprise  the  domain  of  computer  user  satisfaction. 


1 

Bailey,  J.E.  and  Sammy  W.  Pearson,  "Development  of  a  Tool  for  Measuring  and 
Analyzing  Computer  User  Satisfaction,"  Management  Science,  Vol.  29,  No.  5,  May 
1983. 
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Questionnaire  Development  (continued) 

Bailey  and  Pearson  next  borrowed,  from  the  psychological  literature  on  satisfac- 
tion, a  model  that  takes  into  account  both  an  individual's  reaction  to  a  given 
factor  and  the  importance  of  that  factor  to  the  individual  (Wanous  and  Lawler, 
1972).  They  then  applied  a  technique  in  which  adjectives  are  used  to  measure 
perceptions  (Osgood,  et  al,  1962).  Bipolar  adjective  pairs  (e.g.  satisfactory 
vs.  unsatisfactory,  good  vs.  bad)  were  used  to  create  a  scale  on  which  an  indi- 
vidual's perceptions  of  each  factor  could  be  measured.  A  seven  point  scale 
was  used  to  permit  discrimination  in  the  degree  of  "goodness"  or  "badness,"  for 
example,  that  an  individual  attached  to  a  particular  factor.  The  same  method 
was  used  to  measure  an  individual's  perception  of  each  factor's  relative  impor- 
tance or  unimportance. 

An  application  of  this  technique,  to  measure  user  reaction  to  the  "Communication" 
factor,  appears  below.   (This  example  was  included  in  the  modified  version  of 
the  Bailey  and  Pearson  instrument  administered  in  the  User  Satisfaction  Project.) 


Communication;  The  manner  and  methods  of  information  exchange 
between  users  and  computer  services  staff. 


meaningful  meaningless 

precise  vague 

important  unimportant 

A  pilot  test  by  Bailey  and  Pearson  of  their  computer  user  satisfaction  measure- 
ment instrument  resulted  in  an  overall  reliability  coefficient  of  0.95,  which  is 
considerably  greater  than  the  reported  reliability  of  any  other  measurement 
instrument  located  in  the  User  Satisfaction  Project  literature  search.  Bailey 
and  Pearson  also  tested  the  content,  construct  and  external  validity  of  their 
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Questionnaire  Development  (continued) 

instrument,  and  reported  results  that  ranged  from  "high"  (0.91)  to  "satisfactory" 
(0.74). 

The  user  satisfaction  measurement  instrument  located  during  the  literature 
search  is  a  questionnaire  composed  of  thirty  nine  factors,  each  of  which  is 
measured  using  four  adjective  pairs  and  an  importance  rating.  For  the  User 
Satisfaction  Project  two  types  of  modifications  were  made,  the  first  to  tailor 
the  questionnaire  specifically  to  BSO  services,  and  the  second  to  reduce  the 
chance  that,  due  to  its  sophistication  and  length,  the  questionnaire  would  con- 
fuse and/or  alienate  user  agency  respondents. 

Design  modifications  were  made  in  consultation  with  the  Director  of  OMIS  and  BSO 
senior  managers,  and  resulted  in  a  questionnaire  composed  of  fifteen  of  the  ori- 
ginal factors,  organized  under  each  of  six  BSO  service  categories.  One  further 
modification,  the  reduction  in  number  of  adjective  pairs  for  each  factor  from 
four  to  two,  was  made  by  members  of  the  project  team.  The  service  categories, 
factors  and  adjective  pairs  used  in  the  User  Satisfaction  Project  questionnaire 
are  listed  in  Appendix  B  to  this  report. 

Pilot-testing  indicated  that  the  modified  questionnaire  could  be  completed  in 
approximately  twenty  minutes.   Its  subsequent  administration  during  group  inter- 
views (rather  than  by  mail)  not  only  ensured  that  the  greatest  number  of  respon- 
ses would  be  obtained,  but  also  provided  an  opportunity  for  participants  to  seek 
clarification  if  they  were  confused  by  any  aspect  of  the  questionnaire,  thus 
increasing  the  likelihood  of  reliable,  valid  measurement. 


-20- 

Questionnaire  Development  (continued) 

The  modified  questionnaire  proved  to  be  a  highly  reliable  measurement  instru- 
ment. The  split  half  reliability  coefficient  was  computed  for  satisfaction 
ratings  in  each  of  the  six  service  categories,  and  was  found  to  be  0.96  on 
average,  with  a  range  from  0.94  to  0.97.   This  is  a  modest  improvement  over  the 
already  high  reliability  of  the  original  questionnaire. 

Interview  Instrument  Development 

To  facilitate  content  analysis  of  the  information  obtained  during  interviews, 
some  of  the  interview  questions  were  constructed  with  structured  response 
choices,  but  because  a  primary  intent  of  the  interviews  was  to  gather  qualita- 
tive, anecdotal  data,  many  of  the  questions  were  open  ended,  and  participants 
were  encouraged  to  "tell  their  stories." 

Some  of  the  questions  were  designed  to  elicit  information  on  subjects  that  the 
literature  search  had  revealed  might  be  pertinent  (e.g.  user  background). 

A  few  questions  were  designed  to  obtain  participants'  satisfaction  ratings,  as 
well  as  their  comments,  about  certain  BSO  services.  These  ratings  could  later 
be  compared  with  the  more  rigorous  measurement  of  users'  satisfaction  obtained 
using  the  questionnaire  described  above. 

The  group  interview  format  facilitated  information  sharing  among  participants 
within  each  agency,  as  well  as  between  these  agency  participants  and  the  OMIS 
interviewer.  Using  general  guidelines  provided  by  OMIS,  the  selection  of  par- 
ticipants for  each  interview  was  made  by  the  agency  involved. 


-21- 

Questionnaire  Data  Analysis 

The  following  is  a  summary  of  the  procedures  employed  for  coding  and  analyzing 
questionnaire  data.   (BSO  service  categories,  factors  and  adjective  pairs 
included  in  the  questionnaire  are  listed  in  Appendix  B.) 

Within  each  service  category,  user  agency  participants  marked  the  importance  of 
each  of  the  fifteen  factors  on  a  scale  that  was  later  coded  from  one  to  seven. 
The  extreme  ratings  of  one  (on  the  left)  and  seven  (on  the  right)  indicated  most 
importance  and  least  importance,  respectively.  The  intermediate  ratings,  two 
through  six,  indicated  intermediate  degrees  of  importance.  Thus,  the  lower  the 
rating,  the  more  important  the  participant  considered  that  factor  to  be. 

The  middle  or  neutral  position  on  the  scale  was  coded  four.  This  rating  indi- 
cated that  the  associated  factor  was  neither  important  nor  unimportant  to  the 
individual  respondent. 

Participants  could  also  respond  that  a  factor  was  not  applicable  to  them  by 
checking  a  separate  "N.A."  space  on  the  questionnaire  form.  In  such  cases,  and 
in  cases  where  people  omitted  a  rating,  the  importance  score  for  that  factor  was 
treated  as  missing,  for  data  analysis  purposes. 

Mean  (estimated  average)  importance  ratings  for  categories  were  computed  under 
two  separate  rules.  The  first  was  a  generous  rule,  in  which  a  score  was  calcu- 
lated even  if  the  individual  had  rated  only  one  of  the  fifteen  factors  in  the 
category.  The  second  rule  was  a  stringent  one,  in  which  a  mean  importance 
score  was  calculated  only  if  the  respondent  had  marked  importance  for  at  least 
eight  factors,  i.e.  more  than  half  of  the  factors  used  to  describe  the  category. 
All  statistical  analyses  were  run  under  both  rules.  Where  the  test  results  were 
similar  under  the  two  rules,  confidence  increased  that  the  findings  were  not 
spurious. 
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Questionnaire  Data  Analysis  (continued) 

The  procedure  for  calculating  mean  satisfaction  ratings  was  first  to  determine 
the  rating  for  each  of  two  bipolar  adjective  pairs  that  described  each  factor. 
The  rating  for  each  pair  was  determined  in  the  same  manner  as  that  used  for 
importance,  as  described  above.  Each  of  these  adjective  pair  ratings  was  then 
weighted,  using  the  associated  importance  rating.  Finally,  from  the  weighted 
ratings  a  mean  satisfaction  score  was  calculated  for  each  category. 

Again,  both  a  generous  and  a  stringent  inclusion  rule  were  employed.  Under 
the  generous  rule,  the  individual  respondent  only  had  to  have  rated  one  adjec- 
tive pair  and  the  importance  for  that  factor.   Under  the  stringent  rule,  the 
questionnaire  had  to  have  at  least  sixteen  out  of  a  possible  thirty  weighted 
adjective  pair  ratings  in  order  to  be  included  in  the  analysis. 

Interview  Data  Analysis 

An  analysis  was  performed  of  the  qualitative  data  gathered  during  the  group 
interviews.   Comparisons  were  made  among  the  responses  of  participants  in  dif- 
ferent agencies  to  each  of  the  interview  questions;   patterns  were  noted  when 
they  emerged. 

Descriptive  data  were  compiled,  including  data  on  secretariat  affiliation, 
length  of  agency  experience  as  a  BSO  user,  agency  and  individual  experience  with 
other  service  bureaus,  and  participants'  location  in  the  organization.  In  con- 
junction with  questionnaire  data,  these  descriptive  data  were  used  to  test 
hypotheses  about  users'  satisfaction  with  BSO  services. 

Group  ratings  of  satisfaction  with  BSO,  in  general,  and  with  the  BSO  User  Group 
meetings,  in  particular,  were  tabulated  and  converted  to  the  same  seven  ordinal 
positions  used  to  code  questionnaire  responses.   Comparisons  were  then  made  bet- 
ween the  interview  and  questionnaire  ratings. 
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Interview  Data  Analysis  (continued) 

A  list  was  compiled  of  the  suggestions  made  by  those  who  were  interviewed. 

Results  from  the  analyses  of  both  questionnaire  and  interview  data  are  reported 
in  the  Findings  section  of  this  report. 
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Appendix  B 


Table  1 .  BSO  Service  Categories  included  in  the  User  Satisfaction  Project 
Questionnaire. 


Table  2.  Factors  and  Adjective  Pairs  included  in  the  User  Satisfaction  Project 
Questionnaire. 
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Table  1 

BSO  Service  Categories  included  in  the  User  Satisfaction  Project  Questionnaire. 


1 .  Technical  Assistance  in  Problem  Determination  and  Response:  assistance 
with  any  information  systems  problems  that  a  user  has. 


2.  Technical  Assistance  in  Teleprocessing  Applications:  assistance  in  the 
data  communications  area. 


3.  Technical  Assistance  in  Data  Base/File  Management:   consultation  in  the 
use  of  data  base  management  systems. 


k.     Batch  Processing:  regularly  scheduled  production  work;  non-scheduled 
requests. 


5.  On-line  Processing:   inquiry  or  data  entry  input  use  via  on-line 
terminal  device. 


6.  Maintenance-Enhancement  of  Applications:   program  changes. 
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Appendix  C 


Table  3.  User  Satisfaction  Project  Agency  Participants,  1st  Quarter  FY '84 


Table  4.  Agencies'  Mean  Satisfaction  Ratings  for  Six  Categories  of  BSO 
Services,  computed  using  a  generous  inclusion  rule. 


Table  5.  Agencies'  Mean  Satisfaction  Ratings  for  Six  Categories  of  BSO 
Services,  computed  using  a  stringent  inclusion  rule. 


Table  6.  Agencies'  Overall  Usage  and  Mean  Satisfaction  Ratings  for  Two  BSO 
Service  Categories. 
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